Managing File Dependencies Effectively

Follow these guidelines to optimize performance and reliability when using file dependencies.

Note: The service account needs access in Tidal Automation.

Optimize Agent Performance

Avoid overloading Agents: Limit the number of file dependencies per agent. An agent managing more than 200 file dependencies or file events can be overloaded.
Distribute the workload: Distribute file dependencies across multiple agents on different servers to balance CPU and memory usage.
Manage directory size: Keep target directories clean. Large numbers of files in a directory can slow down dependency checks, especially during the reruns.
Minimize wildcard use: Use wildcard cautiously. Scanning entire large directories with wildcard can significantly impact system performance.

Verify file size: Make sure the file size meets expected minimum and maximum requirement to prevent the system from processing empty or unexpectedly large files.
Use the "Stable For DD:HH:MM" condition: Use this condition to ensure the file size remains unchanged for a specific duration (such as five minutes) before the dependency is satisfied. Checking only for a file's existence is often insufficient, as a job might start while the file is still being written.

Note: This condition checks the file's "last modified" timestamp.

Adjust polling intervals: The default polling interval is two seconds. If an agent is overloaded, you can increase this interval to reduce the load.

Note: This is a global setting and affects the responsiveness of all file dependencies.

Use file dependencies for unpredictable schedules: Use these dependencies when file arrival times are inconsistent. This keeps the daily schedule organized and reduces resource overhead.
Inherit agent settings: Select the "Inherit Agent from Job" option to ensure the file check occurs on the same agent that runs the job, guaranteeing the job has access to the required file location.
Combine dependency types: Combine file dependencies with job (e.g., a job completes normally) and variable (e.g., a process flag is set) dependencies for complex workflows. The job runs only when all conditions are met.
Clean up trigger files: If you use "trigger" to signal completion, include a cleanup step to delete the file after a successful run. This prevents the next scheduled run from starting prematurely due to an old file.